Lane Detection for Carla Driving Simulator

For this project we are using dataset based on the Carla (Car Learning to Act) Driving Simulator, an open-source simulator designed for autonomous driving research. Carla simulates driving scenarios that are applicable for training models for tasks like lane detection, object detection, and semantic segmentation.

This data set consists of total samples: 6408 png files = 3075 train + 3075 train_label + 129 val + 129 val_label images.

What is a Receptive Field

The receptive field of a neuron (or unit) in a CNN is the region of the input image that affects the output of that neuron.

If a neuron in a deep layer becomes active), the receptive field tells you which area of the original input image caused it to react.

Why is Receptive Field Important in Vision Tasks?

• Captures Context: Larger RFs help the network understand spatial relationships, objects, and background context.

• Design Consideration: Helps decide kernel sizes, number of layers, and strides when building CNN architectures.

• Segmentation Tasks: Accurate pixel-wise predictions require balancing fine details (small RF) and global context (large RF).

• Deeper Layers = Larger RF: But also need to manage loss of spatial precision.

Receptive Field Growth in CNNs

[Input Image]
   ↓
[Conv Layer 1] → small receptive field (e.g., 3×3)
   ↓
[Conv Layer 2] → medium receptive field (e.g., 7×7)
   ↓
[Conv Layer 3] → large receptive field (e.g., 15×15)

Techniques for Visualizing Receptive Fields

Method | Explanation Gradient-Based | Computes gradients of output w.r.t. input pixels to see which parts affect activation.

Backpropagation | Tracks how changes in pixel values affect activations through layers.

Occlusion Mapping | Slides a gray patch over parts of the input image and sees how output changes.

Increasing Receptive Fields Across Layers

• As you go deeper into the CNN, the receptive field increases.

• Early layers capture edges and textures (small RFs).

• Deeper layers capture shapes, parts, and full objects (large RFs).

Optimizing the Receptive Field

• Kernel Sizes:

Explanation: Kernel size controls the area that each filter covers. A larger kernel size increases the receptive field, allowing the model to capture more context in the image.

Example: Using a 5x5 kernel instead of a 3x3 kernel allows the model to capture broader information, improving its ability to detect lane markings over longer distances.

• Stride:

Explanation: Stride determines how much the filter moves over the input image. Increasing the stride reduces the spatial resolution of the feature maps but increases the effective receptive field.

Example: By using a stride of 2, instead of 1, we can increase the receptive field while reducing the computation required, which is helpful for detecting lanes over long stretches of road.

• Dilation:

Explanation: Dilation expands the filter by spacing the elements, which increases the receptive field without increasing the number of parameters. This allows the network to capture larger contextual information.

Example: A dilated 3x3 convolution can help capture more context compared to a standard 3x3 filter, enabling the model to detect lane markings that span across larger sections of the image.

Image Classification vs Semantic Segmentation

• Classification - assign a label

• Image classification — single label

• Cat or Dog

Image Classification vs Semantic Segmentation

• Semantic Segmentation -> assign a label to each pixel

• Output -> Mask

Convolution Layer

• Typical CNN architecture

Convolution Layer

• Feature Extraction

Convolution Layer - Animation

• Matrix Dot Product

Convolution Layer - Math

• Matrix Dot Product

• Sum of Element-Wise multiplication

• Downsampling

Transposed Convolution

• Opposite of Convolution

• Upsample

• Expand

CNN Encoder-Decoder Architecture

• Convolution Layers in Encoder

• Transposed Convolution Layer in Decoder

U-Net

• Double Convolution Layers

• Transposed Convolution Layer in Expanding Path

• Skip Connections (Concat) — Spatial Context

Model Parameter Count

Training Setup

• PyTorch DataLoaders – Batch training

• Epoch – 10

• Loss – PyTorch CrossEntropy

PyTorch Cross Entropy

• Uses softmax and Negative Log Likelihood internally
• Raw model outputs (logits): [2.5, 1.2, 0.8]
• After softmax: [0.65, 0.18, 0.17]
• If the correct class is index 0, NLL loss = -log(0.65) ≈ 0.43
• If the correct class is index 2, NLL loss = -log(0.17) ≈ 1.77

Training Time

• UNet-3

  ---

Training Time

• UNet-7

  ---

Training Time

• CNN-7

  ---

Training Time

• CNN-5

Model Parameter Count

Comparison of Memory Usage

Performance of Models

• IoU

• Dice Coefficient

Performance of Models

Performance of Models

Top Performers

• CNN-5

• UNet-3

Model Parameter Count

Predictions - CNN-5

Predictions - UNet-3

Conclusion

• CNN-5 > UNet-3

• The UNet-3 performs slightly better than the CNN-5

• Dice score .885 > .85 and the IoU score of .805 > .755

• CNN-5 is 75% smaller than UNet-3